{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Vector and Matrix operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x114b765d0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the matrix is: torch.Size([4, 9])\n",
      "Shape of transposed matrix is: torch.Size([9, 4])\n"
     ]
    }
   ],
   "source": [
    "# torch.arange(start, stop, step) creates a vector whose elements go from\n",
    "# start to stop in increments of step. E.g., torch.arange(0, 72, 8) will \n",
    "# be [0, 8, 16, 24, ..64]. We will create an image with 4 rows and 9 cols\n",
    "# using this function now.\n",
    "I49 = torch.stack([torch.arange(0, 72, 8), torch.arange(64, 136, 8),\n",
    "                torch.arange(128, 200, 8), torch.arange(192, 264, 8)])\n",
    "print(\"Shape of the matrix is: {}\".format(I49.shape))\n",
    "\n",
    "# Transpose of a matrix interchanges rows and cols. A 4 x 9 matrix\n",
    "# becomes 9 x 4 on transposition.\n",
    "I49_t = torch.transpose(I49, 0, 1)\n",
    "print(\"Shape of transposed matrix is: {}\".format(I49_t.shape))\n",
    "\n",
    "# Let us asssert that it is a true transpose, i.e., I[i][j] == I_t[j][1]\n",
    "for i in range(0, I49.shape[0]):\n",
    "    for j in range(0, I49.shape[1]):\n",
    "        assert I49[i][j] == I49_t[j][i]\n",
    "\n",
    "# .T retrieves the transpose of a matrix (array)\n",
    "assert torch.allclose(I49_t, I49.T, 1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dot product\n",
    "The dot product of two vectors $\\vec{a}$ and $\\vec{b}$ represents the\n",
    "component of one vector along the other.\n",
    "\n",
    "Consider two vectors $\\vec{a} = [a_1\\;\\;a_2\\;\\;a_3]$ and\n",
    "$\\vec{b} = [b_1\\;\\;b_2\\;\\;b_3]$.\n",
    "<br>Then $\\vec{a}\\space.\\vec{b} = a_1b_1 + a_2b_2 + a_3b_3$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dot product of these two vectors is: 32\n",
      "Example dot product of orthogonal vectors: 0\n"
     ]
    }
   ],
   "source": [
    "a = torch.tensor([1, 2, 3])\n",
    "b = torch.tensor([4, 5, 6])\n",
    "a_dot_b = torch.dot(a, b)\n",
    "print(\"Dot product of these two vectors is: \"\n",
    "      \"{}\".format(a_dot_b))\n",
    "\n",
    "# Dot product of perpendicular vectors is zero\n",
    "vx = torch.tensor([1, 0]) # a vector along X-axis\n",
    "vy = torch.tensor([0, 1]) # a vector along Y-axis\n",
    "print(\"Example dot product of orthogonal vectors:\"\n",
    "      \" {}\".format(torch.dot(vx, vy)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrix multiplication\n",
    "Matrices can be multiplied with other matrices or vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Matrix vector multiplication\n",
    "Consider a matrix $A_{m,n}$ with m rows and n columns which is multiplied\n",
    "with a vector $\\vec{b_{n}}$ with n elements.\n",
    "\n",
    "Below we show an example with $m = 3$ and $n = 2$.\n",
    "\n",
    "The resultant vector $\\vec{c_{m}}$ is:\n",
    "\\begin{align*}\n",
    "\\begin{bmatrix}\n",
    "        c_{1} \\\\\n",
    "        c_{2}  \\\\\n",
    "        c_{3}\n",
    "\\end{bmatrix}\n",
    "& = \n",
    "\\begin{bmatrix}\n",
    "        a_{11} & a_{12} \\\\\n",
    "        a_{21} & a_{22} \\\\\n",
    "        a_{31} & a_{32}\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "        b_{1} \\\\\n",
    "        b_{2} \\\\\n",
    "\\end{bmatrix}\\\\\n",
    "\\\\\n",
    "c_{1} &= a_{11}b_{1} + a_{12}b_{2} \\\\\n",
    "c_{2} &= a_{21}b_{2} + a_{22}b_{2} \\\\\n",
    "c_{3} &= a_{31}b_{2} + a_{32}b_{2}\n",
    "\\end{align*}\n",
    "\n",
    "In general\n",
    "$$\n",
    "c_{i} = a_{i1}b_{1} + a_{i2}b_{2} + \\cdots + a_{in}b_{n} \\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Xw: torch.Size([15, 1])\n",
      "model output:\n",
      "tensor([[5.1794],\n",
      "        [5.0271],\n",
      "        [6.6973],\n",
      "        [5.2980],\n",
      "        [6.7705],\n",
      "        [6.5460],\n",
      "        [5.1546],\n",
      "        [5.8970],\n",
      "        [5.8990],\n",
      "        [5.8826],\n",
      "        [5.8320],\n",
      "        [5.7036],\n",
      "        [5.7506],\n",
      "        [6.4630],\n",
      "        [5.7754]])\n"
     ]
    }
   ],
   "source": [
    "# Let us consider the familiar cat-brain training dataset\n",
    "# We have defined our model's output to be x.w + b for every\n",
    "# training example x i.e the output is the sum of \n",
    "# dot product of the weight vector with the training input vector \n",
    "# and the bias\n",
    "\n",
    "# We can bulk compute the dot products of all the training\n",
    "# examples with a given weight vector by just multiplying \n",
    "# the matrix X (whose rows correspond to individual training\n",
    "# examples) with vector w.\n",
    "# Finally we add the bias vector. b = 5.0.\n",
    "# Note that X_mul_w is a vector, whereas b is a scalar.\n",
    "# In this case, the scalar is broadcasted\n",
    "# to all elements of the vector.\n",
    "#\n",
    "# Let us reload the cat-brain data matrix X.\n",
    "X = torch.tensor([[0.11, 0.09], [0.01, 0.02], [0.98, 0.91], [0.12, 0.21],\n",
    "              [0.98, 0.99], [0.85, 0.87], [0.03, 0.14], [0.55, 0.45],\n",
    "              [0.49, 0.51], [0.99, 0.01], [0.02, 0.89], [0.31, 0.47],\n",
    "              [0.55, 0.29], [0.87, 0.76], [0.63, 0.24]])\n",
    "w = torch.rand((2, 1)) # a randomly initialized weight vector\n",
    "b = 5.0                      # random bias value\n",
    "X_mul_w = torch.matmul(X, w)\n",
    "\n",
    "# Given the random weight vector and bias, the model will output the\n",
    "# vector model_out (of course this will be very different from the\n",
    "# desired output, we have not chosen the weights and bias optimally.\n",
    "# How to choose optimal will be shown later).\n",
    "model_output = X_mul_w + b\n",
    "\n",
    "print(\"Shape of Xw: {}\\nmodel output:\\n{}\".format(X_mul_w.shape,\n",
    "                                                  model_output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrix Matrix Multiplication\n",
    "Consider a matrix $A_{m,p}$ with m rows and p columns.\n",
    "<br>Let us multiply it with another matrix $B_{p,n}$ with p rows and n columns.\n",
    "<br>The resultant matrix $C_{m,n}$ will contain m rows and n columns.\n",
    "<br>Note that number of columns in the left matrix $A$ should match the number of\n",
    "<br>rows in the right matrix $B$.\n",
    "\n",
    "\\begin{align*}\n",
    "\\begin{bmatrix}\n",
    "        c_{11} & c_{12} \\\\\n",
    "        c_{21} & c_{22} \\\\\n",
    "        c_{31} & c_{32}\n",
    "\\end{bmatrix}\n",
    "& = \n",
    "\\begin{bmatrix}\n",
    "        a_{11} & a_{12} \\\\\n",
    "        a_{21} & a_{22} \\\\\n",
    "        a_{31} & a_{32}\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "        b_{11} & b_{12} \\\\\n",
    "        b_{21} & b_{22} \\\\\n",
    "\\end{bmatrix}\\\\\n",
    "\\\\\n",
    "c_{11} &= a_{11}b_{11} + a_{12}b_{21} \\\\\n",
    "c_{12} &= a_{11}b_{12} + a_{12}b_{22} \\\\\n",
    "c_{21} &= a_{21}b_{11} + a_{22}b_{21} \\\\\n",
    "c_{22} &= a_{21}b_{12} + a_{22}b_{22} \\\\\n",
    "c_{31} &= a_{31}b_{11} + a_{32}b_{21} \\\\\n",
    "c_{32} &= a_{31}b_{12} + a_{32}b_{22}\n",
    "\\end{align*}\n",
    "<br>\n",
    "<br>\n",
    "In general\n",
    "$$\n",
    "\\\\\n",
    "c_{ij} = \\sum_{i=1}^p a_{ip}b_{pj}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A\n",
      "tensor([[1, 2],\n",
      "        [3, 4],\n",
      "        [5, 6]])\n",
      "\n",
      "B\n",
      "tensor([[ 7,  8],\n",
      "        [ 9, 10]])\n",
      "\n",
      "C\n",
      "tensor([[ 25,  28],\n",
      "        [ 57,  64],\n",
      "        [ 89, 100]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "A = torch.tensor([[1, 2], [3, 4], [5, 6]])\n",
    "B = torch.tensor([[7, 8], [9, 10]])\n",
    "C = torch.matmul(A, B)\n",
    "print(\"A\\n{}\\n\".format(A))\n",
    "print(\"B\\n{}\\n\".format(B))\n",
    "print(\"C\\n{}\\n\".format(C))\n",
    "\n",
    "# Dot product is a special case of matrix multiplication\n",
    "w = torch.tensor([1, 2, 3])\n",
    "x = torch.tensor([4, 5, 6])\n",
    "assert torch.dot(w, x) == torch.matmul(w.t(), x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transpose of Matrix Product\n",
    "Given two matrices $A$ and $B$ where the number of columns of $A$ matches\n",
    "<br>the number of rows of $B$, the transpose of their product is the product\n",
    "<br>of the individual transposes in reversed order. $(AB)^T = B^{T}A^{T}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transpose of the product\n",
      "tensor([[ 25,  57,  89],\n",
      "        [ 28,  64, 100]])\n",
      "Product of individual transposes in reverse order\n",
      "tensor([[ 25,  57,  89],\n",
      "        [ 28,  64, 100]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/tx/pqtg9xkd5156pd7hbgdr_4140000gn/T/ipykernel_32801/1962590849.py:9: UserWarning: The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:3281.)\n",
      "  assert torch.all(torch.matmul(A.T, x).T == torch.matmul(x.T, A))\n"
     ]
    }
   ],
   "source": [
    "print(\"Transpose of the product\")\n",
    "print(torch.matmul(A, B).T) \n",
    "print(\"Product of individual transposes in reverse order\")\n",
    "print(torch.matmul(B.T, A.T))\n",
    "\n",
    "assert torch.all(torch.matmul(A, B).T == torch.matmul(B.T, A.T))\n",
    "\n",
    "# This applies to matrix vector multiplication as well\n",
    "assert torch.all(torch.matmul(A.T, x).T == torch.matmul(x.T, A))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix inverse\n",
    "\n",
    "Let us say we want to solve a simultaneuous equation with two variables $x_1$ and $x_2$,\n",
    "<br>Such an equation can be written as\n",
    "\\begin{align*}\n",
    "a_{11}x_1+ a_{12}x_2 &= b_1 \\\\\n",
    "a_{21}x_1 + a_{22}x_2 &= b_2\n",
    "\\end{align*}\n",
    "This can be written using matrices and vectors as\n",
    "$$\n",
    "A\\vec{x} = \\vec{b}\n",
    "$$ where\n",
    "$$\n",
    "A =\n",
    "\\begin{bmatrix}\n",
    "        a_{11} & a_{12} \\\\\n",
    "        a_{21} & a_{22} \\\\\n",
    "\\end{bmatrix}\n",
    "\\space \\space \\space\n",
    "\\vec{x} = \n",
    "\\begin{bmatrix}\n",
    "        x_{1} \\\\\n",
    "        x_{2} \\\\\n",
    "\\end{bmatrix}\n",
    "\\space \\space \\space\n",
    "\\vec{b} = \n",
    "\\begin{bmatrix}\n",
    "        b_{1} \\\\\n",
    "        b_{2} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "Solution of $A\\vec{x} = \\vec{b}$ is \n",
    "$$\n",
    "\\vec{x} = A^{-1}\\vec{b}\n",
    "$$\n",
    "where $A^{-1}$ is the matrix inverse, (assumed $det(A) \\neq 0$).\n",
    "<br>Compare this with the scalar equation $ax = b$ whose solution is $x = a^{-1}b$.\n",
    "\n",
    "The determinant can be computed as  $$det(A) = a_{11}a_{22} - a_{12}a_{21} $$ \n",
    "The inverse is\n",
    "$$\n",
    "A^{-1} = \\frac{1}{det(A)}\n",
    "\\begin{bmatrix}\n",
    "        a_{22} & -a_{12} \\\\\n",
    "        -a_{21} & a_{11}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "Although the above example is shown with a small $2\\times$ system ofsimultaneous equations,\n",
    "<br> the code below is general and works for arbitrary sized linear systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Invertible Matrix\n",
      "Determinant: -1.9999998807907104\n",
      "A:\n",
      "tensor([[2., 3.],\n",
      "        [2., 2.]])\n",
      "\n",
      "A Inverse:\n",
      "tensor([[-1.0000,  1.5000],\n",
      "        [ 1.0000, -1.0000]])\n",
      "\n",
      "Note that determinant of A is -1.9999998807907104 (non-zero),\n",
      "hence A is invertible and hence the equation is solvable\n",
      "\n",
      "Matmul(A, A_inv) is \n",
      " tensor([[ 1.0000e+00, -2.3842e-07],\n",
      "        [ 0.0000e+00,  1.0000e+00]]),which is the identity matrix\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def determinant(A):\n",
    "    return torch.linalg.det(A)\n",
    "\n",
    "def inverse(A):\n",
    "    return torch.linalg.inv(A)\n",
    "\n",
    "# Case1: Invertible Matrix\n",
    "A = torch.tensor([[2, 3], \n",
    "              [2, 2]], dtype=torch.float)\n",
    "\n",
    "A_inv = inverse(A)\n",
    "I = torch.matmul(A, A_inv)\n",
    "\n",
    "# We assert that A A_inv is identity matrix\n",
    "\n",
    "assert torch.allclose(I, torch.eye(2), atol=1e-5) #torch.eye is used to generate identity matrix\n",
    "\n",
    "print(\"Invertible Matrix\")\n",
    "print(\"Determinant: {}\".format(determinant(A))) # 2*2 - 2*3 = -2\n",
    "print(\"A:\\n{}\\n\".format(A)) \n",
    "print(\"A Inverse:\\n{}\\n\".format(A_inv))\n",
    "print(\"Note that determinant of A is {} (non-zero),\\n\"\n",
    "      \"hence A is invertible and hence the equation is solvable\\n\".\\\n",
    "      format(determinant(A)))\n",
    "print(\"Matmul(A, A_inv) is \\n {},\"\n",
    "      \"which is the identity matrix\\n\\n\".format(I))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The torch function torch.eye returns an Identity matrix.\n",
    "I = torch.eye(2)\n",
    "# IA = AI = A\n",
    "assert torch.allclose(torch.matmul(I, A), A, atol=1e-5)\\\n",
    "       and torch.allclose(A, torch.matmul(A,I), atol=1e-5)\n",
    "\n",
    "# AA_inv = A_invA = I\n",
    "assert torch.allclose(torch.matmul(A, A_inv), I, atol=1e-5)\\\n",
    "       and torch.allclose(torch.matmul(A_inv, A), I, atol=1e-5)\n",
    "\n",
    "# aI = Ia = a\n",
    "a = A[0, :]\n",
    "assert torch.allclose(torch.matmul(a, I), a, atol=1e-5) and torch.allclose(torch.matmul(I, a.T), a, atol=1e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Non-Invertible Matrix\n",
      "Determinant: 0.0\n",
      "\n",
      "B tensor([[1., 1.],\n",
      "        [2., 2.]])\n",
      "cannot be inverted\n",
      "because it is a linalg.inv: The diagonal element 2 is zero, the inversion could not be completed because the input matrix is singular.\n",
      "\n",
      "Note that determinant of B is 0.0\n",
      "hence B is non-invertible\n"
     ]
    }
   ],
   "source": [
    "# Case2: Singular Matrix - attempt to invert it directly causes an exception.\n",
    "B = torch.tensor([[1, 1], [2, 2]], dtype=torch.float)\n",
    "print(\"Non-Invertible Matrix\")\n",
    "print(\"Determinant: {}\\n\".format(determinant(B)))\n",
    "try:\n",
    "    B_inv = inverse(B)\n",
    "except RuntimeError as e:\n",
    "    print(\"B {}\\ncannot be inverted\\n\"\n",
    "          \"because it is a {}\\n\".format(B, e))\n",
    "print(\"Note that determinant of B is {}\\n\"\n",
    "      \"hence B is non-invertible\".format(determinant(B)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
